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ABSTRACT 

These  Research  Notes  form  part  of  a  series  of  notes  extracted  from  work  undertaken  by 
Innovation  Science  in  the  establishment  of  Openness  and  Evolvability  assessment 
Methods  and  Processes.  This  set  of  Research  Notes  focusses  on  Documentation  Quality 
Assessment.  This  work  was  undertaken  from  the  late  1990s  to  2007  and  focussed  on  the 
application  to  Submarine  Combat  Systems. 
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1.  Introduction 


These  Research  Notes  have  been  extracted  from  work  undertaken  by  Innovation  Science 
under  contract  to  Defence  Science  and  Technology  Group  during  the  period  from  the  late 
1990s  until  early  2007. 

In  entirety  the  Research  Notes  form  a  subset  of  the  overall  assessment  Methodology  and 
Processes  developed  to  assess  system  level  Openness  and  Evolvability. 

The  Research  Notes  within  this  report  focus  on  Documentation  Quality  Assessment. 


2.  Documentation  Quality  Assessment 

This  process  should  be  used  to  gauge  the  quality  of  documentation  being  assessed  as  part 
of  the  Standards,  Architecture,  Infrastructure  and  Interface  assessment  categories.  The 
document  assessment  considers  either  a  single  or  group  of  related  documents. 

If  a  large  number  of  closely  related  documents  require  assessment,  it  is  acceptable  to  select 
a  random  sample  of  documents  that  fall  into  the  following  three  categories,  [1]: 

•  the  seven  largest  documents  in  the  set 

•  the  seven  smallest  documents  in  the  set 

•  seven  documents  of  average  size. 


If  fewer  than  21  documents  exist  in  the  set,  then  a  thorough  assessment  should  consider 
them  all. 

Many  of  the  document  assessment  metrics  result  in  the  allocation  of  a  "maximum  score" 
as  well  as  a  "penalty  point"  score.  The  maximum  score  applies  to  the  overall  document 
quality  score.  However,  the  penalty  point  system  is  calculated  for  each  individual 
document,  regardless  of  the  number  of  documents  that  combine  to  form  the  set  of 
documentation.  Each  of  the  penalty  point  scores  are  considered  after  first  evaluating  each 
of  the  documents  in  the  set. 

If  a  maximum  score  is  encountered,  that  score  is  the  best  score  that  can  be  awarded  as  the 
documentation  quality  score  for  the  documentation  being  assessed.  The  assessment  should 
continue  however,  as  the  final  score  may  ultimately  be  lower  than  the  interim  maximum 
score. 

The  flowcharts  shown  in  Figure  1  and  Figure  2  summarise  the  assessment  process.  Each 
question  within  the  flowchart  is  explained  in  greater  detail  in  the  sections  that  follow. 
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Figure  1.  Documentation  Quality  Assessment  Flowchart  (1  of  2) 
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Figure  2.  Documentation  Quality  Assessment  Flowchart  (2  of  2) 

2.1  Documentation  Quality  Assessment  Questions 

2.1.1  Does  documentation  consist  of  more  than  one  physical  document? 

If  more  than  one  document  combines  to  form  the  documentation  being  assessed,  the  set  of 
documents  need  to  be  assessed  for  their  consistency,  completeness  and  the  ability  to 
readily  navigate  between  the  documents  in  the  set. 

2.1.2  Is  consistency  maintained  across  all  documents  in  set? 

Focusing  only  on  the  overall  style,  layout  and  presentation  of  the  documents,  are  the 
documents  consistent?  A  lack  of  consistency  between  documents  can  lead  to  confusion, 
and  adversely  affects  usability  and  documentation  maintenance  [2] . 

2.1.3  Are  all  documents  available  and  complete? 

If  it  is  clear  that  documents  are  missing  from  the  set,  the  assessment  must  immediately 
apply  the  lowest  score  to  the  documentation  set.  Missing  documentation  is  likely  to  make 
it  virtually  impossible  for  a  third-party  to  independently  use  the  remainder  of  the 
documentation  to  independently  achieve  the  same  interpretation  as  the  original  vendor. 

2.1.4  Is  navigation  between  documents  clear? 

If  it  appears  as  though  the  set  of  documents  are  complete,  but  it  is  unclear  how  the 
documents  in  the  set  relate  to  one  another,  then  a  third-party  will  find  it  difficult  to  ensure 
they  locate  sufficient  information  to  achieve  the  same  interpretation  as  the  original  vendor. 
The  document  set  is  therefore  marked  poorly. 
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2.1.5  Do  unassessed  documents  remain  in  set? 

If  multiple  documents  are  being  assessed  as  a  single  set,  and  more  documents  exist  that 
have  not  yet  been  checked,  then  progress  to  2.9.1. 6  (section  2.1.6);  otherwise,  skip  to 
2.9.12.20  (section  2.1.20).  Note  that  if  a  large  number  of  documents  combine  to  form  the 
set,  only  a  random  subset  of  the  documents  require  assessment,  provided  an  equivalent 
number  of  random  samples  are  selected  from  the  smallest,  largest  and  average  sized 
documents  within  the  set. 

2.1.6  Select  a  document  that  has  not  yet  been  assessed. 

Choose  a  previously  unassessed  document  from  the  set  of  documents  to  begin  a  detailed 
assessment. 

2.1.7  Initialise  penalty  points  for  document  to  zero. 

Penalty  points  may  be  accumulated  by  each  document  during  the  assessment  process. 
Each  assessment  criteria  adds  either  zero,  one  or  two  penalty  points  to  a  single  document's 
assessment.  Once  each  of  the  assessment  criteria  have  been  considered  for  a  given 
document,  the  total  number  of  penalty  points  should  be  recorded  so  that  it  can  be 
considered  later  in  the  overall  document  quality  assessment. 

2.1.8  Any  issues  with  spelling,  grammar  or  punctuation? 

Read  randomly  selected  paragraphs  drawn  from  a  cross-section  of  the  document  and 
judge  the  accuracy  of  the  document's  spelling,  grammar  and  punctuation.  Incorrect 
punctuation  or  grammar  can  make  the  document  cumbersome  and  difficult  to  read,  and 
can  sometimes  lead  to  multiple  interpretations.  While  considerable  latitude  needs  to  be 
given  to  permit  different  writing  styles,  this  assessment  is  primarily  concerned  with 
identifying  issues  with  spelling,  grammar  or  punctuation  that  introduce  ambiguity  or  that 
make  the  document  difficult  to  read. 

In  terms  of  punctuation,  focus  should  be  directed  towards  the  correct  use  of  full  stops, 
commas,  semi-colons,  colons,  dashes  and  parentheses.  Both  underuse  and  overuse  should 
be  noted,  although  an  occasional  indiscretion  should  be  overlooked  unless  it  gives  rise  to 
ambiguity.  Sentences  that  do  not  adequately  resolve  ambiguity  should  be  flagged  as 
significant  risks. 

For  example,  consider  the  following  ambiguously  constructed  sentence: 

"In  the  event  of  a  discrepancy,  the  message  shall  be  sent  to  Goliath,  the  central  server  and 
main  message  repository." 

Unless  the  reader  already  has  intimate  knowledge  of  the  system  architecture,  it  is  quite 
conceivable  that  the  sentence  could  be  interpreted  with  one  of  at  least  four  meanings: 

1.  The  message  should  be  sent  to  "Goliath",  which  is  the  central  server  and  main 
message  repository 
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2.  The  message  should  be  sent  to  both  "Goliath"  (which  is  the  central  server),  and  the 
message  repository 

3.  The  message  should  be  sent  to  "Goliath"  as  well  as  to  the  central  server  and 
message  repository  (which  are  one  and  the  same) 

or 

4.  The  message  should  be  sent  to  "Goliath"  as  well  as  to  both  the  central  server  and 
message  repository. 

2.1.9  Any  issues  with  sentence  structure? 

Poorly  chosen  sentence  structure  reduces  readability.  This  metric  is  closely  related  to  the 
use  of  punctuation,  but  is  essentially  focused  on  determining  whether  or  not  sentences 
within  the  document  follow  expected  rules.  The  documentation  being  assessed  is  usually 
technical  or  academic  in  nature.  Hence,  journalistic  tendencies  (such  as  fragmented 
sentences  that  are  sometimes  included  for  dramatic  effect)  are  usually  inappropriate. 

Issues  to  look  out  for  include: 

1.  Dependent  clauses:  Two  or  more  dependent  clauses  written  as  separate  sentences 
that  would  be  more  easily  understood  if  they  were  revised,  and  possibly  also 
rearranged,  to  form  a  single  sentence  with  appropriate  punctuation.  [3]  [4] 

2.  Fragmentation:  Sentences  without  a  main  verb  or  subject.  [4] 

3.  Independent  clauses:  A  sentence  containing  two  or  more  independent  phrases  that 
have  not  been  connected  using  an  appropriate  dependent  marker,  or  that  would  be 
more  easily  understood  if  they  were  written  as  separate  sentences.  [5] 

2.1.10  Is  the  layout  and  style  clear  and  consistent? 

Inconsistent  layout  and  style  makes  it  difficult  to  navigate  the  documentation.  The  implicit 
hierarchy  of  information  within  the  documentation  may  also  be  lost  if  consistency  is  not 
maintained. 

Evaluate  the  document  for  consistency  in  terms  of: 

•  font  (size  and  typeface) 

•  use  of  emphasis  (font  size,  weight,  colour,  italics  and  underline) 

•  paragraph  alignment  (left,  right,  centre  or  fully  justified) 

•  indentation 

•  heading  styles  and  numbering 

•  use  of  footnotes  and  side  bars 

•  figure,  table  and  equation  captioning 

•  overall  page  layout  (including  headers  and  footers). 
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2.1.11  Are  illustrations  relevant,  consistent  and  clear? 

Diagrams,  illustrations  and  figures  can  offer  a  rapid  method  to  convey  complex 
information.  However,  they  should  aim  to  provide  support  to  a  related  block  of  text  rather 
than  to  exist  in  place  of  a  textual  explanation  [6].  Furthermore,  visuals  should  be 
functional,  consistent  and  clear  in  order  to  achieve  greatest  value  to  the  reader. 

Evaluate  the  document  for  consistency  between  illustrations  (where  appropriate).  Look  for 
clear  linkages  between  the  text  and  associated  visuals.  Determine  if  the  visuals 
successfully  enhance  the  accompanying  text  or  if  they  have  been  inserted  in  isolation. 
Consider  the  complexity  of  the  visuals:  Are  supplementary  visuals  used  to  explain  or 
detail  parts  of  a  higher-level  visual,  or  is  too  much  detail  contained  in  a  single  visual? 

2.1.12  Is  translation  to  English  clear? 

This  question  is  relevant  only  if  the  document  has  been  translated  from  another  language. 
Translated  technical  documents  risk  introducing  ambiguities  and  multiple  interpretations 
in  the  English  version  that  were  not  present  in  the  native  language  version.  The  complex 
and  non-linear  correlations  between  words  of  different  languages  make  the  task  of 
accurately  translating  documentation  particularly  difficult.  Regardless  of  this  difficulty, 
the  fact  remains  that  a  poorly  translated  document  introduces  a  substantial  risk  that  the 
translated  document  will  be  misinterpreted  by  native  speakers  of  the  resulting  language. 

2.1.13  Is  vocabulary  consistent  and  glossary  complete? 

Many  technical  documents  will  introduce  abbreviations  and  acronyms  to  avoid  repeating 
long  strings  of  words  throughout  a  document.  It  is  important  that  these  terms  are  both 
appropriately  defined  when  they  are  first  used  within  the  document,  and  (more 
importantly  for  reference  documents1)  included  within  a  glossary  at  an  easily  located 
position  within  the  document. 

2.1.14  Do  quality  Table  of  Contents  and  Index  exist? 

Although  this  question  literally  asks  for  the  existence  of  a  Table  of  Contents  and  Index,  a 
penalty  should  be  applied  if  the  quality  of  these  elements  is  poor.  Hence,  even  if  both  a 
Table  of  Contents  and  Index  exist  within  the  document,  if  the  quality  of  one  or  both  is 
poor,  the  metric  should  be  scored  as  if  the  poor  quality  elements  are  missing. 

Easy  navigation  within  the  document  is  enhanced  through  the  presence  of  both  a  Table  of 
Contents  and  Index.  Most  word  processing  and  publishing  programs  offer  straight¬ 
forward  methods  to  automatically  generate  a  Table  of  Contents.  However,  constructing  a 
usable  Index  usually  requires  good  planning  and  some  manual  effort. 

A  sufficient  Table  of  Contents  would  include  a  number  of  heading  levels  that  is 
appropriate  to  the  complexity  of  the  document.  If  the  document  is  particularly  complex. 


1  Reference  documents  are  often  not  read  in  a  specific  order.  Relying  on  an  acronym  definition 
being  present  at  its  first  use  within  the  document  may  make  it  difficult  for  a  reference  user  to  locate 
the  appropriate  meaning  of  the  acronym.  Reference  documents  should  therefore  always  contain  a 
glossary  of  such  terms 
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with  many  heading  levels,  the  use  of  an  Index  to  supplement  a  shorter  Table  of  Contents 
would  be  preferable  over  the  blanket  construction  of  a  very  long  Table  of  Contents 
containing  many  low-level  headings.  However,  a  complex  document  with  only  top-level 
headings  in  its  Table  of  Contents  would  be  equally  inappropriate. 

Before  evaluating  the  Index,  attempt  to  identify  several  topics  derived  from  reading  other 
parts  of  the  document,  and  search  for  those  topics  using  the  index.  A  good  quality  index 
should  point  to  the  introduction  of  the  topic  as  well  as  supplementary  usage  (if 
appropriate).  Ideally,  the  index  should  emphasise  (e.g.  bold)  the  page  numbers  of  the  most 
significant  references  for  common  topic  words.  The  index  should  also  include  inverted 
references  and  sub-entries  to  index  topics  that  are  often  described  using  multiple  words 
(e.g.  "ethernet  network"  should  also  be  listed  under  "network"  with  a  sub-entry, 
"ethernet"). 

The  following  set  of  rules  have  been  provided  for  guidance  purposes  only.  These  rules 
have  been  derived  from  usability  testing  [7],  and  are  included  within  this  assessment  as  an 
example  of  how  a  good  quality  index  could  be  constructed.  Other  methods  may  be  equally 
appropriate,  and  the  assessor  should  make  a  judgement  as  to  the  success  of  the  index  in 
the  context  of  the  document's  target  domain  and  audience. 

Suggested  rules  for  creating  quality  indexes  [7]  are  summarised  in  the  following  list. 

•  Indexes  should  include  chapter  titles,  concepts,  proper  names,  terms,  titles, 
relationships,  and  subheadings  as  entries 

•  Indentation  should  be  used  instead  of  run-on  styles 

•  Long  entries  that  wrap  over  multiple  lines  should  be  avoided 

•  Main  entries  should  include  sub-entries  for  all  items  discussed  under  the  main 
entry  topic  except  for  titles  and  proper  names 

•  Main  entries  should  avoid  having  specific  page  references  when  sub-entries  exist 
for  the  main  entry 

•  Main  entries  should  nominate  page  spreads  (e.g.  p50-55)  to  identify  areas  of 
primary  discussion  for  the  main  entry  topic 

•  Sub-entries  should  be  present  for  any  topic  that  has  a  large  number  (5  or  more)  of 
page  references 

•  Main  entries  should  exist  for  detailed  discussions  continuing  for  10  pages  or  more 

•  Prepositions  and  conjunctions  should  be  avoided  within  sub-entries 

•  Sub-entries  should  also  be  listed  as  top-level  entries  (double-posting) 

•  Acronyms  should  be  included  in  the  index 

•  Sub-entries  should  be  phrased  to  avoid  commas,  where  possible  (e.g.  "test  data 
creation"  rather  than  "test  data,  creating"). 
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2.1.15  Is  history  of  change  management  evident? 

Documents  that  include  a  well-maintained  change  history  give  confidence  to  the  reader 
that  the  information  contained  within  the  document  is  up-to-date.  The  change  history  may 
or  may  not  include  a  summary  of  topics  affected  by  each  change,  but  should  include  (as  a 
minimum)  the  release  date  and  version  number  of  the  resulting  document.  Depending  on 
the  type  of  document  (how  technical  and  its  intended  audience),  change  bars  may  also 
enhance  reader  confidence  levels. 

If  a  change  history  is  present,  look  for  evidence  that  it  has  been  kept  up-to-date. 

2.1.16  Are  internal  and  external  references  complete  and  accurate? 

Accurate  cross-referencing  is  critical  to  the  user  being  able  to  quickly  navigate  the 
document  (or  set  of  documents).  Missing  or  inaccurate  cross-references  greatly  decrease 
usability. 

References  should  include: 

•  Page  and/  or  section  cross-references  where  there  is  a  clear  need  to  relate  two 
sections  of  the  document 

•  Page  and/  or  section  cross-references  to  common  areas  of  the  document  to  avoid 
duplication  of  common  information 

•  A  list  of  closely  related  documents  (particularly  when  the  document  forms  part  of  a 
set) 

•  Bibliographical  references  to  previously  published  documents. 


2.1.17  Has  IV &V  been  conducted  on  document? 

If  Independent  Verification  and  Validation  (IV&V)  has  been  performed  on  the  document, 
the  assessment  can  assume  the  IV&V  activities  addressed  the  technical  validity  of  the 
document.  If  evidence  of  IV&V  has  been  performed,  go  to  2.1.18  (section  2.1.18); 
Otherwise,  consider  the  document  in  greater  detail  via  question  2.1.19  (section  2.1.19). 

2.1.18  Has  document  passed  IV&V? 

An  external  assessment  of  the  document  should  be  able  to  be  distilled  down  to  a  binary 
pass/ fail  mark.  Use  the  independent  verification  and  validation  assessment  to  determine 
whether  or  not  the  document  has  achieved  its  technical  aim. 

2.1.19  Is  language  clear  and  unambiguous? 

An  independent  verification  and  validation  exercise  has  not  been  conducted  on  the 
document.  However,  it  is  unlikely  that  the  openness  assessment  will  have  the  time  or 
resources  to  conduct  such  an  activity.  Hence,  the  assessor  should  randomly  select  a 
chapter  of  the  document  (or  equivalent)  and  gauge  the  ease  with  which  they  can 
understand  the  contents  based  on  the  language  used  and  whether  or  not  the  meaning 
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appears  ambiguous.  Consideration  should  be  given  to  the  intended  audience  and  the  level 
of  assumed  knowledge  implied  by  such  an  audience  —  particularly  when  the  assessor  is 
not  an  expert  in  the  subject  area. 

2.1.20  Penalty  Points  for  any  document  >=  10? 

If  any  of  the  penalty  point  totals  derived  for  individual  documents  in  the  set  is  equal  to  ten 
(10)  or  more,  the  document  is  deemed  to  contain  serious  flaws  and  is  awarded  the  lowest 
score. 

2.1.21  Penalty  Points  for  any  document  >=  6? 

If  any  of  the  penalty  point  totals  derived  for  individual  documents  in  the  set  is  between  6 
and  9,  the  document  is  deemed  to  contain  too  many  flaws  to  be  sufficient  for  a  third-party 
to  use  the  document  in  isolation  of  the  author,  with  confidence  that  they  will  correctly 
understand  and  interpret  its  content. 

2.1.22  Penalty  Points  for  any  document  >-21 

If  any  of  the  penalty  point  totals  derived  for  individual  documents  in  the  set  is  between  2 
and  5,  the  document  is  flagged  as  requiring  improvement,  but  is  still  more  than  likely 
sufficient  to  allow  a  third-party  to  use  the  document  in  isolation  of  the  author  and 
maintain  a  consistent  interpretation  of  its  content. 

If  the  penalty  point  totals  are  all  less  than  2  (either  zero  or  one),  any  issues  with  the 
documentation  are  considered  minor  and  are  unlikely  to  pose  any  risk  to  third-party 
interpretation  of  the  document  contents. 
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